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We investigate a recurrent neural network model with common external and bias inputs 
that can retrieve branching sequences. Retrieval of memory sequences is one of the most 
important functions of the brain. A lot of research has been done on neural networks that 
process memory sequences. Most of it has focused on fixed memory sequences. However, 
many animals can remember and recall branching sequences. Therefore, we propose an as- 
sociative memory model that can retrieve branching sequences. Our model has bias input 
and common external input. Kawamura and Okada reported that common external input 
enables sequential memory retrieval in an associative memory model with auto- and weak 
cross-correlation connections. We show that retrieval processes along branching sequences 
are controllable with both the bias input and the common external input. To analyze the 
behaviors of our model, we derived the macroscopic dynamical description as a probability 
density function. The results obtained by our theory agree with those obtained by computer 
simulations. 

KEYWORDS: branching sequence, common external input, bias input, associative memory, 
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1. Introduction 

Retrieval of memory sequences is one of the most important functions of the brain. There 
have been many studies about neural networks that process memory sequences, In these 
models, cross-correlation between one pattern and the next one in the sequence is embedded 
in synaptic connections. These connections make the model retrieves the sequential memory 
deterministically. The stored sequences are confined to fixed sequences without branching 
points. However, many animals, including humans, memorize and retrieve memory sequences 
with branching points. A specific example observed in songbirds, which have attracted in- 
terest as model animals for sequence learning and generation. Bengalese finches sing songs 
with finite state syntax that are composed of chunks of song notes and branching points of 
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note sequences. Because such song patterns can be considered to be rudiments of human lan- 
guage syntax, and thus it is important to investigate the neural mechanisms for storing and 
retrieving branching sequences. Guyon et al. treat branching sequences with various synaptic 
transmission delays.^ In their model, however, the state transitions are deterministic, depend- 
ing on neural activity history. This makes it hard to dynamically select the next state at a 
branching point and to transit stochastically. 

Aoyagi and Aoki^'^ have investigated a model with auto- and weak cross-correlation con- 
nections and showed that transitions between attractors can be invoked by synchronous spikes. 
Kawamura and Okada assumed synchronous spikes to be correlated noises, i.e., common ex- 
ternal input, and theoretically investigated the dynamics of associative memory models with 
common external input. ^'^ In this paper, we propose an associative memory model with com- 
mon external input and bias input that can store and retrieve branching sequences. Each 
memory pattern is stored as an attractor in an auto-correlation connection. Sequences among 
the attractors are stored in weak cross-correlation connections. Common external input can 
induce state transitions along the stored sequences. Since the distances among all memory 
patterns are equal, the model is trapped in a mixed state of memory patterns at a branching 
point. We introduce bias input to transit from the mixed state to the desired memory state. 
Due to common external input, the states of neurons are correlated. Therefore, the model has 
sample dependence. The behavior of such a model can be described by a probability den- 
sity function of order parameters, i.e. ovcrlap.^^'^^'^^ We derive a probability density function 
(PDF) of macroscopic state variables and discuss the properties of our model. 

This paper is organized as follows. In the next section, we propose a model with common 
external input and bias input. In §3, we discuss a model without bias input and present 
behavioral results for the model obtained theoretically and by computer simulations. In §4, 
we discuss a model with bias input. We show that bias input enables our model to transit to 
a target memory pattern. The final section concludes the paper. 



We consider an associative memory model consisting of N neurons. The state of the ith 
neuron takes = ±1 at time t and is updated synchronously by 



where the output finction is F{h) = sgn(/i), and is a synaptic connection from the jth 
neuron to the ith neuron. ^* is independent external input for the ith neuron at time t. 'xf is 
common external input, which affects all neurons similarly. 

B\ is also external input and is correlated with a particular memory pattern. Since this 



2. Model 
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external input has an effect that biases the state toward a target pattern at a branching point 
in the sequences, we call it bias input. Bias input is generated by 

Proh[Bi = ±l]=^^f^, (2) 

where 

J2bt<^, (3) 

and is overlap between bias input and the memory pattern at time t. The constant, 
c, in (1) is the parameter that determines amplitude of the bias input. 

The element of the memory patterns = ■ ■ ■ , ^jv)"^ takes ±1 with 

Prob[Cf = ±l] = i. (4) 

In this paper, we consider two sequences of memory patterns: sequence B, which is a typical 
branching sequence, and sequence A, which is a sub-sequence of sequence B (Fig. 1). Synaptic 
connection J^j is given by 

/i=l 1/=! 

The number of patterns following pattern is denoted as p^. The (//, z^)-th component of a 
matrix A is determined according to 

1, if /i = 1/ 

V = ^ s/P., if/i/i^, , (6) 

0, if /X 7^ u, e e 

Matrix A becomes a transition matrix. For example, in the sequence shown in Fig. 1(a), the 
numbers of the patterns become pi = 3, and P2 = Ps = P4 = 0, and the matrix becomes 

/ 1 \ 



(7) 



£/3 1 
£/3 1 
\ e/3 1 J 

where a parameter £ is e ^ 1, which determines the strength of the cross-correlation connec- 
tion. With A^iy = 5fj_i,, the model corresponds to an autoassociative memory model, whereas 
with Afj^i, = Sn^i^^i, it corresponds to a sequential associative memory model. Kawamura and 
Okada's model is represented as A^j^j, = 5^,^ + sSfj^^^+i.^*^ 

We define the overlap as the direction cosine between the state of neurons, a;* at time t, 
and the memory pattern, 

N 



N 

i=l 
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Prom eq. (1), (5), and (8), the state of the neurons, xj'^^ becomes 

p p 

N 



f' = ^ EEV«-^^..+C* + r/*+cS|) . (9) 



, /i=i 1^=1 

The initial state is determined according to 

Prob[x° = ±l] = i^p^. (10) 
Therefore, the overlap between pattern and initial state is mo- 

2. 1 Effect of bias input 

Here, we explain the role of bias input 5*. We define h\ in internal potential as 

^NEEv^M- (11) 

When sequence A is stored, from eq. (5) and eq. (7), 



p 

£ 



If cc* that is, ml ^ 1, then hj becomes 

h^-^^ + li^+^ + e)- (13) 

The common external input, therefore, will make the state transit to the mixture state sgn(^^+ 
+ ^^).-'^'^ Since the mixture state correlates with memory patterns and the model 

has the ability to transit to those three patterns. However, with N ^ oo, the distances between 
the mixed state and the three memory patterns are equal. This symmetry prevents the model 
from transiting to any memory pattern. In this state, the bias input that correlates with a 
given memory pattern can break the symmetry and can induce a transition to the target 
memory pattern (Fig. 2). The value of for the target pattern could be any appropriate 
small value capable of breaking the symmetry. 
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Fig. 2. Schematic diagram of branching state transition mechanisms induced by common external 
input and bias input, x-marks denote mixed state of patterns and 



3. No bias input model (c = 0) 

3.1 Theory 

First, we discuss the model without bias input, i.e., c = 0. In this case, our model can be 
regarded as a generalized model of Kawamura and Okada's one with A^,^ = 6^1, + eS^^iy+i.^^ 
To describe the behavior of the model using macroscopic variables, we consider the thermo- 
dynamic limit N ^ oo. For the sake of simplicity, we assume that the number of memory 
patterns is finite, p~ 0{1). Independent external input is time independent of each neuron 
and is assumed to obey the Gaussian distribution, where the mean equals and the variance 
equals a^. The common external input is also time independent, and is assumed to obey 
the Gaussian distribution, where the mean equals and the variance equals S^. 

Due to the common external input, the states of the neurons are correlated, so sample 
dependence must be taken into account. ^^"^^ We derive a probability density function (PDF) 
of macroscopic variables m^. When 77* is known at given time t and N ^ 00, m'^j^i^ii = 
1,2, could be given as the function of mt = {ml,mf, ■■,m^) and 77*, 

-. N / p p \ 

m>t^,{mt,rf) = j^Y.^lF ^ A.^^rmf + C| + ry* (14) 

i=l \y=l p=l J 

= l^j D^eF [y.^ A^^^m', +az + rf^ ^ (15) 
where Dz = exp {^^^ ^''^d (•)^ denotes the average over the whole memory pattern 
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The function erf(?x) is defined as 

erf (tx) = — / dtexp (-t^) . (17) 
V'^ Jo 

Next, when ry* obeys the Gaussian distribution: 

the distribution of overlap mt can be described as the PDF, p {mt, if ) . Since rf is independent 
of mt, p {mt, rf ) can be divided into two PDFs, 

p{mt,rf) =p{mt)p{rf) . (19) 

Therefore, the PDF of mt is given by 

P p p 
p (mt+i) = / n drrftdrfp {mt) p {rf) J] ^ (K+i - K+i ("^t> rf)) , (20) 

where S{-) denotes the Dirac delta function. We evaluate (20) using the Monte Carlo method 
in the following section. 

3.2 Results 

Focusing on branching state transitions, we mainly treat sequence A in Fig. 1(a). Figure 3 
shows a sample of time evolution of the overlap by (16), where £ = 0.1, a = 0.1, and S = 0.37. 
The initial overlap is mo = 1.0. Numbers in Fig. 3 denote indexes of the memory patterns. 
Because there is no bias input, mj,mf, and mf behave in a similar fashion. Therefore, no 
memory pattern is retrieved completely; only a mixed state is retrieved. 

Figure 4 shows a sample of the time evolution of overlaps calculated by computer simu- 
lation, where e = 0.1, a = 0.1, S = 0.37, and the number of neurons is N = 100,000. The 
transition to the mixed state occurs after the pattern In this sample, even though there 
is no bias input, pattern is completely retrieved. The memory patterns and can also 
be retrieved in other samples. 

Since the states of neurons have sample dependence due to common external input, we 
investigate the distributions of the overlaps. To evaluate the probability of a given memory 
pattern being retrieved, we calculate the marginal probability density functions of m^*, /i = 
1, 2, 3, and 4, from the joint probability density function p {mt) in eq. (20), 

pK) = / lldm'^p{mt). (21) 

Figure 5 shows the marginal PDFs of the overlaps, p {m}), p (mf), p {mf), and p (m^). The 
probability densities at = 1.0, = 1.0, and = 1.0 are zero. That is, the patterns 
and are not retrieved. Figure 6 shows histograms obtained by computer simulation. 
The frequencies at = 1.0, = 1.0, = 1.0 are observed to a certain extent. The 
disagreement between computer simulations and theory may be due to the finite size effect. 
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2 + 3+4 



Fig. 3. Time evolutions of overlaps ml, ml, mf , and m| obtained theoretically for bias input (c = 0), 
where e = 0.1, a — Q.l, and 6 — 0.37. 




Fig. 4. Time evolution of overlaps ml,mf, m,f, and mf when there is no bias input (c = 0), obtained 
by computer simulations {N = 100,000), where e = 0.1, a = 0.1, and 6 = 0.37. State transition 
occurs from pattern to 



which breaks the symmetry between the mixed state and the memory patterns. 

4. Bias input model (c > 0) 

4-1 Theory 

Next, we discuss the model with bias input, where m^^DM = 1)2, ...,p are given by the 
function of rrit = {ml,m^, ■■,mf) and 77*, 

?nf+i(mt,??*) 

= ^J2^fF{h' + C! + v' + cBj) (22) 

i=l 

Z)^^^|i±^^iM!F ^h' + az + 7j' + c] + 1 - ^^=1 ^t^'^ F [h' + az + r]'~c] |^ 

(23) 
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where 

h' ^Y.Y.A,,i^ml (25) 

v=l p=l 

When the memory pattern ^ is biased {h^ > 0, 6^ = 63 = 64 = 0, and c > 0) in network 
storing sequence A, m^_^^(mt, r/*) for // / 2 become 



and for /x = 2, 
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The effect of the bias input appears in the second term of RHS in (27). Therefore, becomes 
larger than mf and mf. The PDFs of overlaps m^, fi = 1, 2, are also given by the form 
ofeq. (20). 

4-2 Results 

We have shown that the model can transit to one of the memory states, i.e., transfer out 
of a mixed state. The parameters related to the bias input are = 0.1, bj = bf = bf = 0, and 
c = 0.05. That is, the bias input, whose overlap with the pattern is bf = 0.1, is multiplied by 
0.05. Other parameters are £ = 0.1, a = 0.1, and S = 0.37. Figure 7 shows the time evolutions 
of the overlaps mj,mf,mf, and mf calculated by (24). Biased pattern can be retrieved in 
most samples. Figure 8 shows marginalized PDFs p {m}) , p (mf ) , p (mf ) , and p [mf) at time 
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Fig. 7. Time evolution of overlaps obtained theoretically in model with bias input (6| = 0.1, c = 0.05), 
where e = 0.1, a = 0.1, and 5 = 0.37. 



t. Probability density at m? = 1.0 increases with time. Figure 9 shows histograms obtained by 
computer simulations {N = 100, 000). The model transits to state = 1.0 or = 1.0 with 
small probability. Except for this small disagreement, the results agree with those obtained 
theoretically in Fig. 8. 

We discussed the model in which a common external input obeys the Gaussian distri- 
bution. We can, however, choose a certain common external input. We found empirically 
that a particular common external input caused a state transition. For instance, the common 
external input 

1 , t mod 50 = 
r?* = < 0.6 , <t mod 50 < 3 , (28) 
, otherwise 

v 

makes the state transit along sequence B. To store sequence B, transition matrix A becomes 

000000e\ 

1 

1 

e/3 1 

£ 1 

£ 1 

£ 1 

£ £ £ 1 

Only memory pattern ^ is biased (6^ = 0.2, c = 0.05). In this case, m^^i, = 1,2, ...,8 are 
given by eq. (27) for limit iV — oo. Figure 10 shows state transitions of model storing sequence 
B. It shows the time evolutions of the overlaps obtained by (a) computer simulation [N = 
100, 000) and (b) theoretical calculation and (c) represents the particular common external 



/ 1 

e/3 
e/3 



A 









(29) 



10/14 



J. Phys. Soc. Jpn. 



Full Paper 




Fig. 8. PDFs of overlaps obtained theoretically in model with bias input (6| = 0.1, c = 0.05), where 
£ = 0.1, a = 0.1, and 5 = 0.37. 

input given in cq. (28). The parameters are e = 0.1, a = 0.1, 62 = 0.2, and c = 0.05. With 
the combination of such common external and bias inputs, state transitions with branching 
are controllable. 

5. Conclusion 

We proposed an associative memory model that retrieved branching sequences using com- 
mon external input and bias input. Since the model has sample dependence due to the common 
external input, we derived the macroscopic description as a probability density function of 
macroscopic variables. Then, we discussed the dynamics of the retrieval processes and effects 
of the bias input. The results obtained theoretically agree with those obtained by computer 
simulations. 

As a result, in the limit of A/" — 00, the model without the bias input transits to a mixed 
state at the branching point in the sequence. We found that the model with bias input, on 
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the other hand, could transit to a target memory pattern. When this happens, the overlap 
between the bias input and the biased memory pattern, 6^, only needs to be large enough to 
break the symmetry of the mixed state. We also showed that state transition in branching 
sequences can be invoked by bias input and common external input, which takes a particular 
time structure, as shown in Fig. 10. 

The transition mechanism that uses common external input is consistent with an 
experiment in songbirds, which suggests that the song control nucleus HVC receives common 
external input at the beginning of each song note.^^ As mentioned above, Bengalese finches 
sing songs with finite-state automata and branching points.^ The state transition mecha- 
nism we have shown could play a role in the nervous system of Bengalese finches. Thus, 
our model has the advantage of being experimentally testable in an existing biological system. 
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Fig. 10. Time evolution of overlap when model storing sequence B is obtained by (a) computer 
simulations {N = 100, 000) and (b) theoretical calculation and (c) represents the common external 
input given in eq. (28), where £ = 0.1, tr = 0.1, 61 = 0-2, and c = 0.05. 
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